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Assessment  of  Interactive  Multimedia  Instruction  (IMI) 
Courseware  Can  Enhance  Army  Training 


Since  1998,  the  Army’s  Training  and  Doc¬ 
trine  Command  (TRADOC)  has  been 
engaged  in  establishing  and  fielding  The 
Army  Distributed  Learning  Program 
(TADLP)  to  enhance  and  extend  traditional 
methods  of  learning.  The  Army  intends  to 
achieve  a  number  of  important  goals  through 
distributed  learning  (DL),  including  increased 
access  to  standardized  training,  improved  unit 
operational  readiness,  and  reduced  costs.  The 
Army  envisages  a  greatly  increased  role  for  DL 
over  time,  and  the  development  of  interactive 
multimedia  instruction  (IMI)  courseware  is  an 
important  element  of  the  training  strategy. 

Development  and  evaluation  of  Army  DL 
is  decentralized  in  individual  proponent  schools 
and  centers,  and  there  have  been  limited  efforts 
to  assess  the  effectiveness  of  DL  training  at 
the  program  level.  TRADOC  asked  Arroyo  to 
assess  how  efficiently  and  effectively  TADLP 
has  accomplished  its  objectives  overall.  For  one 
component  of  this  evaluation,  the  research  team 
developed  and  tested  a  method  of  evaluating  the 
instructional  design  and  technical  features  of 
asynchronous  IMI  courses.  Using  standards  from 
the  training  and  development  community,  the 
team  developed  criteria  to  evaluate  IMI  course¬ 
ware.  The  researchers  then  applied  the  criteria  to  a 
sample  of  79  lessons  from  10  high-priority  courses 
in  order  to  assess  the  feasibility  of  this  approach 
for  evaluating  courseware  in  a  highly  resource- 
constrained  environment,  illustrate  the  kinds  of 
information  produced  by  such  an  evaluation,  and 
demonstrate  how  that  information  can  be  used  to 
identify  areas  for  improvement  in  courseware  and 
to  monitor  quality  at  the  program  level. 

Some  Features  of  IMI  Courseware  Need 
Improvement 

An  example  of  the  study  results  can  be  found  in 
the  table  on  the  following  page. 


Key  Points 

•  Distributed  learning  (DL)  is  the  key  to  the 
Army's  training  strategy,  but  there  are  no 
systematic  program-level  assessments  of  DL 
effectiveness. 

•  RAND  Arroyo  Center  developed  an  approach 
to  evaluating  DL  courseware  that  reveals 
strengths  and  needs  for  improvement  in  tech¬ 
nical  features  and  instructional  design. 

•  This  method  is  cost-effective  and  should  be  part 
of  a  comprehensive  evaluation  program  sup¬ 
porting  continuous  improvement  in  Army  DL. 


Analysis  revealed  that  technical  characteristics 
were  the  strongest  features  of  the  courseware.  All 
courses  were  easy  to  navigate,  and  cues  to  the 
learner’s  position  in  the  course  were  readily  acces¬ 
sible.  The  key  areas  for  improvement  in  technical 
features  are  (1)  ensuring  that  students  can  launch 
the  courseware  without  professional  assistance 
and  (2)  linking  course  content  with  supplemen¬ 
tary  instructional  resources.  Providing  direct 
access  to  reference  materials  such  as  glossaries 
and  field  manuals  could  give  students  powerful 
tools  for  rapidly  deepening  their  knowledge  in 
specific  task  areas. 

Production  quality  was  generally  strong 
(see  the  table).  Narration  was  easy  to  understand, 
courses  had  minimal  irrelevant  content,  and 
graphics  and  text  were  legible.  Improvement  is 
needed,  however,  in  eliminating  sensory  con¬ 
flicts,  such  as  simultaneous  presentation  of  text 
and  spoken  narration,  and  in  the  enhanced  use 
of  multimedia. 

Ratings  of  pedagogical  characteristics  revealed 
a  number  of  strengths,  including  clear  lesson 
objectives,  appropriate  sequencing  of  lessons, 
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Production-Quality  Criteria  for  Courseware 


Criterion 

Rating 

Legibility  of  text  and  graphics 

0.80 

Audiovisuals 

Narration  easy  to  understand 

1.00 

Minimal  irrelevant  content 

0.85 

Use  of  animation/video  to  demonstrate  process 

0.75 

Techniques  to  maintain  learner  interest 

0.50 

Few  sensory  conflicts 

0.40 

■  85-100%  rated  positive.  B70-84%  rated  positive.  ■<  70%  rated  positive. 

clear  and  comprehensive  instruction  of  concepts,  and  oppor¬ 
tunities  for  learners  to  correct  their  strategies  in  checks  on 
learning.  However,  pedagogy  was  the  area  most  in  need  of 
improvement.  A  pervasive  problem  was  a  lack  of  context 
or  examples  from  job  or  mission  environments.  Courses 
also  need  to  do  a  better  job  on  instruction  of  procedures  by 
providing  clearer  demonstrations,  offering  higher-fidelity 
opportunities  for  practice,  and  including  explanations  of  why 
procedures  work  the  way  they  do. 

Best  Practices  for  DL  Training 

The  results  suggest  that  IMI  is  best  suited  for  training  con¬ 
cepts  and  processes,  but  can  be  used  to  train  procedures  in 
some  situations: 

•  When  procedures  can  be  practiced  realistically  within 
the  context  of  IMI,  such  as  completing  forms,  or  with 
the  addition  of  simple  job  aids. 

•  When  learning  is  not  subject  to  rapid  decay  or  is  easily 
refreshed. 

•  When  IMI  supplements  resident  training. 

•  When  training  is  supported  by  a  high  level  of  instructor- 
student  interaction. 


The  Army  also  can  improve  the  quality  of  instruction 
and  increase  user  engagement  by  designing  IMI  with  higher 
levels  of  interactivity  between  the  student  and  the  course¬ 
ware.  For  example,  IMI  that  requires  students  to  move 
objects  on  the  screen  can  be  used  to  train  procedures  such 
as  using  a  compass.  For  more  complex  tasks,  such  as  how  to 
enter  and  clear  a  building,  videogame-like  simulations  could 
be  used  in  which  learners  must  make  decisions  about  appro¬ 
priate  methods  of  entry  in  a  dynamic  environment. 

The  Method  Can  Contribute  to  Program-Level 
Assessments  of  Training  Effectiveness 

The  method  employed  by  the  Arroyo  research  team  provides 
a  systematic  method  of  evaluation  using  a  comprehensive  set 
of  criteria  based  on  standards  proposed  by  training  experts.  It 
yields  quantifiable  data,  enabling  integration  of  results  across 
courses,  schools,  and  other  units.  It  requires  relatively  mod¬ 
est  resources.  By  applying  the  method  to  a  larger  and  more 
diverse  set  of  courses  on  an  ongoing  basis,  the  Army  could 
gain  valuable  information  about  courseware  quality,  identify 
needs  for  improvement,  and  monitor  the  effects  of  changes  to 
training  policy,  development  processes,  or  doctrine. 

In  addition  to  evaluating  courseware,  a  comprehensive 
evaluation  of  training  quality  requires  several  other  types  of 
measures  and  methods,  including  (1)  measures  of  outcomes 
(student  reactions,  learning,  job  performance,  and  organiza¬ 
tional  outcomes);  (2)  test  evaluation  to  assess  the  quality  of 
course  tests;  and  (3)  administrative  data,  such  as  completion 
rates,  cost  data,  and  cycle  time  of  courseware  production, 
which  can  point  to  potential  negative  or  positive  aspects  of 
course  quality.  Taken  together,  these  measures  would  provide 
a  basis  for  achieving  continuous  improvement  in  the  develop¬ 
ment  and  use  of  IMI  and  help  the  Army  reach  its  strategic 
goals  for  DL.  ■ 
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